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Cost-Benefit  Study  of  a  Project  to  Lower  Cost  and 
Improve  Fleet  Readiness  through  Integrating  the 
Management  of  Technical  Information 


Dan  Levine —  Dr.  Dan  Levine  holds  a  PhD  in  physics  and  an  MA  in  economics.  After  a  time  at 
Harry  Diamond  Labs  and  the  National  Bureau  of  Standards,  he  has  been  working  since  1962  on 
economics  and  cost-effectiveness  studies  at  the  Center  for  Naval  Analyses  and  the  Institute  for 
Defense  Analyses.  His  areas  of  research  have  been  in  Navy  and  Army  logistics,  and  a  variety  of 
areas  in  OSD  planning.  Levine  gives  the  lecture  on  cost  effectiveness  analysis  for  the  course  in  cost 
analysis  that  the  IDA  Cost  Analysis  and  Research  Division  gives  annually  to  graduate  students  in 
operations  research  at  George  Mason  University. 


Abstract 

This  paper  describes  a  cost-benefit  analysis  by  the  Institute  for  Defense  Analyses  of 
the  “Bridge  Project”  that  ADL  (Advanced  Distributed  Learning)  is  conducting  for  the  Office  of 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (OSD(AT&L)  to  improve  the 
management  of  Integrated  Logistics  Support  (ILS).  The  Project  is  part  of  the  OSD  RTOC 
program  (Reduction  in  Total  Ownership  Cost).  The  Bridge  Project  focuses  on  integrating 
(Bridging)  the  management  and  production  of  technical  manuals  and  training  courses.  The 
benefits  would  be  lower  cost  to  produce  these  manuals  and  courses  in  the  future,  and 
improved  readiness  through  insuring  the  delivery  of  consistent  and  up-to-date  logistics 
support  to  the  Fleet. 

Manuals  and  courses  are  currently  produced  by  entirely  separate  processes.  Tech 
writers  and  course  developers  obtain  contractor  data  on  systems  and  equipments  in  parallel, 
they  express  the  information  in  different  formats,  they  organize  the  data  in  different 
structures,  and  they  store  the  data  in  different  repositories.  Cost  is  therefore  higher  because 
of  duplication  of  resources  and  the  difficulties  in  re-using  data.  The  lack  of  integration  can 
also  reduce  readiness,  since  it  opens  up  the  possibility  that  the  tech  manuals  and  training 
courses  present  disparate  information,  thus  depriving  ship  operators  and  maintainers  of  the 
most  effective  support. 

The  Bridge  Project  seeks  to  relieve  these  problems  by  designing  new  software, 
technical  and  business  processes  to  integrate  the  production  of  technical  manuals  and 
training  courses.  All  technical  and  learning  content  would  be  expressed  by  the  same  digital 
specification  (the  S1000D  industry  specification),  they  would  employ  the  same  structure 
(Data  Modules),  and  the  data  would  all  be  stored  in  the  same  repositories  (Common  Source 
Data  Bases,  or  CSDBs).  The  project  is  developing  an  API  (Application  Programming 
Interface)  to  enable  course  developers  to  exchange  data  with  any  CSDB,  and  a  Web 
Service  to  more  quickly  update  tech  manuals  and  training  courses  in  response  to 
Engineering  Change  Proposals  (ECPs). 

The  analysis  finds  that  the  Bridge  would  achieve  net  benefits  (benefits  less  costs)  of 
approximately  $87  million  over  10  years,  far  more  than  enough  to  cover  the  $8.7  million  10- 
year  cost  of  producing  all  Navy  HM&E  (Hull,  Mechanical  and  Electrical)  technical  manuals 
and  training  courses  delivered  by  Navy  e-Learning,  a  part  of  the  Naval  Education  and 
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Training  Command  (NETC).  A  sensitivity  analysis  of  the  most  uncertain  inputs  yields  a 
range  of  results  (net  benefits)  from  $32  million  to  $166  million. 

The  Bridge  could  also  contribute  to  shipboard  readiness  by  insuring  the  Navy’s  policy 
of  providing  up-to-date  and  consistent  information  to  the  Fleet  upon  installation  of  new 
systems  and  equipment.  A  parametric  analysis  indicates  that  by  increasing  the  availability  of 
the  electronic,  ordnance  and  HM&E  components  of  a  single  new  DDG  1000  destroyer  for 
only  a  single  day  would  increase  effectiveness  the  Navy  values  at  $2  million. 

Executive  Summary 

The  Institute  for  Defense  Analyses  (IDA)  is  conducting  a  cost-benefit  analysis  of  the 
“Bridge  Project”  that  the  Advanced  Distributed  Learning  (ADL)  office  is  conducting  for  the 
Office  of  the  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (OSD(AT&L)). 
The  project  is  part  of  OSD’s  RTOC  program  (Reduction  in  Total  Ownership  Cost).  The 
project’s  focus  is  on  improving  how  the  Navy  manages  several  aspects  of  Integrated 
Logistics  Support  (ILS)  in  order  to  reduce  the  cost  of  producing  technical  manuals  and 
training  courses.  A  related  benefit  is  increasing  readiness  through  insuring  the  Navy’s  policy 
of  having  the  appropriate  logistics  support  on  hand  when  new  systems  and  equipment 
upgrades  are  fielded. 

Under  current  ILS  management,  manuals  and  courses  are  produced  by  entirely 
separate  processes.  Tech  writers  and  course  developers  obtain  contractor  data  on  new 
systems  and  equipment  upgrades  in  parallel.  They  express  the  data  in  different  formats, 
they  organize  the  data  in  different  structures,  and  they  store  the  data  in  different 
repositories.  The  cost  of  producing  logistics  support  is  therefore  higher  because  of  the 
duplication  of  resources  and  the  difficulties  in  re-using  data.  The  lack  of  integration  can  also 
reduce  readiness,  since  it  opens  up  the  possibility  that  the  tech  manuals  and  training 
courses  present  disparate  information.  And  there  may  be  delays  in  updating  the  information 
in  response  to  Engineering  Change  Proposals  (ECPs).  These  disparities  and  delays  can 
deprive  ship  operators  and  maintainers  of  the  most  up-to-date  information  on  their  systems 
and  equipment. 

The  Bridge  project  is  designing  a  new  process — new  software,  technical  and 
business  processes  to  integrate  (“Bridge”)  the  production  of  technical  manuals  and  training 
courses.  The  initial  beneficiary  of  the  funding  is  the  Littoral  Combat  Ship  (LCS)  Mission 
Modules  Program  (PMS  420),  which  is  integrating  the  Mission  Modules  into  the  LCS.  Under 
the  Bridge  Project,  all  technical  and  learning  content  would  be  expressed  by  the  same  digital 
specification  (the  S1000D  industry  specification),  organized  by  the  same  structure  (Data 
Modules),  and  stored  in  the  same  repositories  (Common  Source  Data  Bases).  The  project  is 
also  developing  an  Application  Programming  Interface  (API)  to  grant  all  course  developers 
access  to  any  CSDB,  and  a  Web  Service  to  quickly  identify  the  technical  and  learning 
content  that  must  be  reviewed  for  updating  in  response  to  Engineering  Change  Proposals 
(ECPs). 

The  cost-benefit  study  estimates  the  investment  and  implementation  costs  of 
designing  and  implementing  the  integrated  approach.  Investment  is  measured  by  the 
personnel  and  related  expenses  of  the  project  during  the  second,  or  coming  year  of  this  2- 
year  project.  (The  first-year  costs  are  sunk,  and  no  longer  relevant.)  Implementation  involves 
training  technical  writers  and  course  developers  in  using  the  Bridge,  and  the  licenses  and 
user  fees  to  cover  the  additional  costs  of  maintaining  the  networks  and  the  data  repositories. 
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The  Bridge  Project’s  benefits  are  the  anticipated  reduction  in  the  costs  of  producing  future 
technical  manuals  and  training  courses,  and  possible  improvements  in  shipboard  readiness. 

The  cost  savings  were  estimated  by  first  listing  the  dozens  of  tasks  (38  for  the 
manual  and  80  for  the  courses)  to  produce  a  nominal  500-page  technical  manual  and  a 
nominal  one-content-hour  training  course.  Project  personnel  estimated  the  number  of  staff 
hours  to  produce  these  tasks  under  both  current  and  Bridge  processes,  and  thus  the  staff 
hour  savings  from  using  the  Bridge.  Pay  rates  are  used  to  convert  the  staff  hour  savings  to 
cost  savings.  The  final  step  is  scaling  up  the  results  to  the  sample  of  yearly  production  of 
technical  manuals  and  training  courses.  Costs  and  benefits  were  expressed  in  10-year 
present  values  calculated  using  the  2.4%  annual  discount  rate  mandated  by  OMB  for  10- 
year  studies. 

Two  different  samples  were  chosen  for  analysis,  reflecting  the  different  perspectives 
of  OSD  and  the  PMS  420  Program  Office.  The  first  analysis  recognizes  OSD’s  interest  in 
seeing  whether  the  new  software  and  technical  and  business  processes  that  comprise  the 
Bridge  would  lead  to  positive  net  benefits  to  DoD  overall — would  the  benefits  cover  the  costs 
if  implemented  by  the  Navy  and  other  Services  as  a  whole.  This  analysis  is  therefore 
conducted  for  a  substantial  number  of  the  Navy’s  yearly  production  of  technical  manuals 
and  training  courses:  all  Hull,  Mechanical,  and  Electrical  (HM&E)  technical  manuals 
produced  by  the  Naval  Ship  System  Engineering  Station  (NAVSSES)  in  Philadelphia  (over 
45,000  pages  annually),  and  50%  of  the  Computer-Based  Training  (CBT)  courses  delivered 
by  Navy  e-Learning  (NeL),  a  part  of  the  Naval  Education  and  Training  Command  (NETC) 
(approximately  3,300  content  hours  annually,  in  total). 

Only  50%  of  the  NeL  courses  were  considered  because  our  analysis  of  the  course 
titles  indicated  that  only  half  of  the  courses  trained  “hard  skills.”  These  are  the  training 
courses  that  deal  with  equipment  and  other  technical  content,  and  whose  cost  would 
therefore  be  reduced  by  integrating  the  production  of  training  courses  and  technical 
manuals.  (Although  courses  that  train  “soft  skills”  such  as  leadership  and  personal 
advancement  would  not  be  directly  affected  by  integration,  they  might  benefit  from  the 
information-organizing  features  of  the  other  Bridge  innovations.) 

The  second  analysis,  reflecting  the  Program  Office’s  perspective,  is  a  test  case  of 
the  results  of  the  aggregate  analysis  in  which  the  Bridge  is  applied  to  a  particular  system, 
the  LCS’s  AN/AQS-20A  mine  hunting  sonar.  The  focus  here  is  on  the  benefits  alone — 
whether  the  Bridge  would  lead  to  reductions  in  future  cost  of  producing  future  technical 
manuals  and  training  courses  for  this  system.  It  would  not  be  reasonable  to  expect  the 
benefits  for  a  single  program  to  cover  the  full  investment  and  implementation  costs  of  the 
Bridge,  which  could  lead  to  savings  across  DoD. 

The  first  analysis  finds  that  the  Bridge  would  save  approximately  $87  million  in  10 
year  cost  in  producing  the  Navy  HM&E  manuals  and  50%  of  NeL  delivered  courses — far 
more  than  enough  to  cover  the  $8.7  million  investment  and  implementation  costs  of  the 
program.  The  second  analysis  finds  that  the  Bridge  would  produce  substantial  savings  of 
almost  $306  thousand  in  producing  technical  manuals  and  training  courses  for  the  LCS 
AN/AQS-20A. 

Dealing  with  uncertainty  was  a  major  analytical  problem.  Although  much  of  the 
analysis  used  historical  data  from  NAVSSES,  NETC  and  the  AN/AQS-20A  program,  the  cost 
savings  are  also  based  on  several  uncertain  inputs  relating  to  the  new  Bridge  process.  A 
sensitivity  analysis  was  conducted  of  five  inputs:  1)  the  future  pay  rate  for  technical  writers 
and  course  developers  who  are  trained  in  using  the  Bridge,  2)  the  percentage  of  training 
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hours  that  would  be  benefited  by  the  Bridge,  3)  the  investment  cost  (to  hedge  against 
unanticipated  cost  of  developing  the  Bridge),  4)  the  implementation  cost  (to  hedge  against 
problems  caused  by  the  cultural  changes  in  Navy  programming),  and  5)  the  percentage 
saving  in  course  developer  staff  hours  from  using  the  Bridge.  Considering  these  changes  in 
combination  yielded  a  full  range  of  net  benefits  (benefits  less  costs)  for  the  aggregate  case 
varying  from  a  minimum  of  $32  million  to  the  Base  Case  of  $78  million  ($87  million  less  $8.7 
million)  to  a  maximum  of  $160  million.  Efforts  will  be  taken  in  the  second  year  of  this  study  to 
further  refine  the  inputs. 

There  is  every  reason  to  expect  that  net  benefits  would  be  significantly  larger  if  the 
Bridge  were  applied  to  all  Navy  manuals  and  courses,  and  those  of  the  other  Services  as 
well. 


The  final  benefit  for  analysis  was  the  improvement  in  Fleet  readiness.  Integrated 
production  of  technical  manuals  and  training  courses  would  increase  the  likelihood  of 
providing  the  Fleet  with  up-to-date  and  consistent  information,  thus  providing  some 
insurance  to  the  Navy’s  policy  of  fielding  new  systems  and  equipment  upgrades  only  when 
the  appropriate  logistics  support  is  available.  A  parametric  analysis  indicates  that  increasing 
the  availability  of  the  electronic,  ordnance  and  HM&E  components  of  a  single  new  DDG 
1000  destroyer  by  a  single  month  would  yield  a  gain  in  effectiveness  the  Navy  values  at  $2 
million. 
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Topics 


I  DA 


■  Current  problems  with  managing  data  for  Navy  technical 
manuals  and  training  courses 

■  “Bridge”  project  to  relieve  these  problems  by  integrating 
(“Bridging”)  the  production  of  technical  manuals  and 
training  courses 

■  IDA  cost-benefit  analysis  of  the  Bridge 

■  Methodology  for  estimating  the  benefits 

■  Quantitative  results 

■  Sensitivity  analysis 

■  Further  work 
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Current  Problems  with  Managing  Data  for 
Technical  Manuals  and  Training  Courses 


I  DA 


■  Content  management  processes  are  non- 
integ  rated 

■  Tech  writers  and  course  developers  obtain  OEM 
contractor  data  in  parallel 

■  Data  are  stored  in  different  formats 

■  Data  are  stored  in  different  repositories 

■  Course  developers  use  different  tools 

■  No  systematic  way  to  identify  data  requiring 
modification  by  equipment  upgrades  via  ECPs 
(Engineering  Change  Proposals) 
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Bridge  Project  to  Relieve  Problems 
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■  Proposal  to  integrate  (“Bridge”)  the  management 
of  data  for  technical  manuals  and  training 
courses 

-  Project  is  funded  by  OSD(AT&L) 

-  Part  of  OSD  RTOC  program  (Reduction  of  Total 
Ownership  Costs) 

■  What  are  the  expected  costs  and  benefits  of 
integration? 

-  Are  the  benefits  substantial? 

-  Do  they  cover  the  costs? 
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Main  Features  of  the  Bridge 


Production  of  Technical  Manuals  and 
Training  Courses 


Current  Processes 

Bridge 

Content  management 

Non-integrated 

Integrated 

OEM  data 

Obtained  in  parallel 

Obtained  once 

Data  format 

Various 

Common — SI  000D 
industry  specification 

Data  repository 

Various 

Common — CSDBs 

Course  developer 
authoring  tools 

Various 

Various,  but  with 
common  link  (API)  to 
CSDBs 

Identification  of  data 
requiring  modification 
by  ECPs 

No  systematic  method 

Single  tool  (Web 
Service) 
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Cost  Benefit  Analysis  of  the  Bridge 
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■  Costs 

-  Investment 

•  personnel  cost  of  project 

-  Implementation 

•  training  future  tech  writers  and  course  developers  to  use  the  Bridge 

•  Change  in  site  licenses  and  user  fees  to  maintain  CSDBs 

■  Benefits 

-  Reduction  in  future  cost  of  technical  manuals  and  training 
courses 

-  Improvement  in  shipboard  readiness  through  timely  delivery  of 
information  (parametric  analysis) 

■  Calculate  10-year  costs  and  benefits  (present  values) 
using  2.4%  annual  discount  rate  mandated  by  OMB 
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Estimating  the  Benefits 


I  DA 


■  Analyze  a  nominal  500-page  technical  manual 
and  nominal  one-content-hour  training  course 

-  Identify  the  detailed  tasks 

-  Estimate  the  staff  hours  to  perform  these  tasks  using 
current  processes  and  the  Bridge 

-  Calculate  Bridge  savings  (5.3%  and  15.3%  for  manuals 
and  courses) 

-  Use  pay  rates  to  convert  staff  hour  savings  to  cost 
savings 

■  Scale  up  the  savings  to  two  samples  of  Navy 
manuals  and  courses 
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Two  Samples  for  Analysis  of  Savings 
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■  OSD  perspective 

-  Reduction  in  total  ownership  costs 

-  Would  the  benefits  cover  the  costs  for  all  Navy  (and 
DoD)  manuals  and  courses? 

-  Apply  Bridge  to  a  large  sample:  all  HM&E  (Hull, 
Mechanical  and  Electrical)  manuals  and  all  NeL 
(Navy  e-Learning)  delivered  courses 

■  LCS  Program  Office  perspective 

-  Would  integration  save  money  for  a  single  system? 

-  Apply  Bridge  to  LCS  mine  hunting  sonar  (AN/AQS- 
20A) 
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Examples  of  Staff-Hour  Changes  Using  the  Bridge 


Staff  Hours 

Increase  in  cost 

Draft  new  business  rules  to  integrate  technical 

data  and  training 

60 

Coordinate  technical  data  with  training  activities 

44 

Reduction  in  cost 

Greater  data  reuse 

160 

Web  Service  to  modify  technical  information  in 

response  to  ECPs 

280 

Use  Web  Service  to  update  links  with  training 

product 

72 

Develop  storyboards  for  training  courses 

16 

Publish  content  in  Learning  Management 

Systems 

11 

Quantitative  Analysis:  Base  Case 
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10-Year  Benefits 
(Present  Value) 

OSD  Perspective  (HM&E/NeL) 

Costs  (investment,  implementation) 

$8.7  million 

Benefits  (future  savings) 

$86.8  million 

Net  benefit  (benefits  less  costs) 

$78.1  million 

LCS  Perspective  (mine  hunting  sonar) 

Benefits 

$305.7  thousand 

I  DA 


Sensitivity  Analysis 


■  Study  uses  historical  data 

-  Production  of  HM&E  technical  manuals  (NAVSSES) 

-  Annual  content  hours  of  Navy  e-Learning  training  courses  (NETC) 

-  Manuals  and  Courses  for  mine-hunting  sonar  (NTSP) 

■  But  the  savings  also  depend  on  uncertain  inputs 


Net  Benefits 


Base  Case  Variation 


VahaU  ons  that  lower 

net 

:  benefits 

$1.8  million 

100%  higher  investment  cost: ; ; ;  |:|||||^ 

$  3.6  million 

200%  higher  implementation  cost 

$6.9  million 

$  20.7  million 

50%  lower  training  course 

cost  savings 

1 13% 

7.65°J  I 

Variations  that  raise 

net 

benefits 

50%  higher  hourly  pay  rate 

$65 

$97 

50%  higher  hard 

skills  rate 

50% 

75% 

10 


$  Millions 


Results  of  Sensitivity  Analysis 

(Net  Benefits) 
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Applying  the  Bridge  to  all  Navy  (&  DoD)  manuals  and  courses 

would  increase  net  benefits 
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Further  Work 


I  DA 


■  Validate  inputs  during  2nd  year  of  study 
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BACKUPS 


Results  of  Sensitivity  Analysis: 

OSD  Perspective 
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10-Year  Net  Benefits 
(Present  Values) 

Variations  that  lower  net  benefits 

100%  higher  investment  costs 

$76  million 

200%  higher  implementation  costs 

$64  million 

7.65%  training  course  savings  (vice  15.3%) 

$48  million 

Variations  that  raise  net  benefits 

50%  higher  pay  rate 

$120  million 

75%  hard  skills  (vice  50%) 

$109  million 

Base  Case 

$78  million 

Minimum  (all  variations  that  lower  net  benefit) 

$32  million 

All  changes 

$77  million 

Maximum  (all  higher  net  benefit) 

$166  million 

Increasing  sample  to  all  Navy  (DoD?)  manuals  and  courses 

would  increase  net  benefits 


Improvement  in  Readiness 
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■  Recent  Navy  policy:  after  shipyard  availabilities,  don’t 
deploy  ships  with  new  systems  and  equipment  upgrades 
until  all  logistics  support  is  in  place 

■  Effect  of  the  Bridge 

-  Earlier  completion  of  logistics  support  (manuals  and  courses) 

-  Allow  ships  to  be  deployed  sooner 

■  Parametric  analysis:  Permitting  a  single  DDG-1000  to 
deploy  a  single  month  earlier  would  increase 
effectiveness  of  electronic,  ordnance  and  HM&E 
components  the  Navy  values  as  $2  million. 
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